DOCUMENT RESUME 



ED 443 805 



SP 039 362 



AUTHOR 

TITLE 



INSTITUTION 

SPONS AGENCY 
ISBN 

PUB DATE 
NOTE 

CONTRACT 
AVAILABLE FROM 



PUB TYPE 
EDRS PRICE 
DESCRIPTORS 



Teitel, Lee 

Assessment: Assessing the Impacts of Professional 
Development Schools. AACTE Professional Development School 
Practice Series. 

American Association of Colleges for Teacher Education, 
Washington, DC. 

Department of Education, Washington, DC. 

ISBN- 0-89333 -179-1 
2000 - 00-00 

7 7p . ; Written with Ismat Abdal-Hagg. 

ED - 9 9 - CO - 0 0 0 7 

AACTE Publications, 1307 New York Avenue, N.W. , Suite 300, 
Washington, DC 20005-4701; Tel: 202-293-2450; Fax: 
202-457-8095; Web site: http://www.aacte.org ($10). 

Guides - Non-Classroom (055) -- Tests/Questionnaires (160) 

MF01/PC04 Plus Postage. 

*College School Cooperation; Elementary Secondary Education; 
Evaluation Methods; Higher Education; Partnerships in 
Education; Preservice Teacher Education; *Prof essional 
Development Schools; *Program Effectiveness; *Program 
Evaluation; Research Methodology 



ABSTRACT 



This booklet is designed to help Professional Development 
School (PDS ) planners and implementers who seek guidance in designing and 
carrying out impact documentacion . It can also help students, researchers, 
and policymakers interested in investigating PDS effectiveness. The booklet 
offers a conceptual framework for conducting impact documentation, a 
checklist for assessing existing documentation efforts or guiding those in 
the planning stages, and descriptions of existing impact documentation 
activity at a number of sites. A key component is the collection of 
instruments, protocols, and analytical approaches developed by several 
partnerships. There are seven chapters: (1) ’’The Challenges and Importance of 

PDS Impact Documentation"; (2) "A Conceptual Framework for Assessing PDS 
Impacts"; (3) "Designing and Implementing Impact Research: Suggested Steps"; 
(4) "Tool Kit for Analyzing Impact Research Plans and Implementation"; (5) 
"Profiles of Selected PDS Impact Research Efforts"; (6) "Sample Protocols, 
Questionnaires, and Analytic Approaches"; and (7) "References and Additional 
Resources." (Contains 27 references.) (SM) 



Reproductions supplied by EDRS are the best that can be made 
from the original document. 



ASSESSMENT 

Assessing the Impacts 
of 

Professional Development Schools 




LEE TEITEL 

* ,4 \ With 

j| | lll | W ABDAL-HAQO 

Lv- 



■ § mm 



<■ 

■-it • 



U.S. DEPARTMENT OF EDUCATION 
Office of Educational Research and Improvement 

EDUCATIONAL RESOURCES INFORMATION 
CENTER (ERIC) 

□ This document has been reproduced as 
received from the person or organization 
originating it. 

□ Minor changes have been made to 
improve reproduction quality. 



Points of view or opinions stated in this 
document do not necessarily represent 
official OERI position or policy. 




'-EKIC 



AACTE 

fessional Development School Practice Series 



2 



Assessing the Impacts 
of 

Professional Development Schools 

Lee Teitel 
with 

Ismat Abdal-Haqq 



AACTE 

Professional Development School Practice Series 



The American Association of Colleges for Teacher Education is a national, 
voluntary association of colleges and universities with undergraduate or 
graduate programs to prepare professional educators. The Association sup- 
ports programs in data gathering, equity, leadership development, network- 
ing, policy analysis, professional issues, and scholarship. 

The opinions, conclusions, and recommendations expressed in this mono- 
graph do not necessarily reflect the views or opinions of the American Asso- 
ciation of Colleges for Teacher Education or the ERIC Clearinghouse on Teach- 
ing and Teacher Education. AACTE and the Clearinghouse do not endorse 
or warrant this information. AACTE and the Clearinghouse are publishing 
this document to stimulate discussion, study, and experimentation among 
educators. The reader must evaluate this information in light of the unique 
circumstances of any particular situation and must determine independently 
the applicability of this information thereto. 

This project has been funded, at least in part, with Federal funds from the 
U.S. Department of Education under contract number ED-99-CO-0007. The 
content of this publication does not necessarily reflect the views or policies 
of the U.S. Department of Education nor does mention of trade names, 
commercial products, or organizations imply endorsement by the U.S. 
Government. 

Assessing the Impacts of Professional Development Schools may be ordered from: 

AACTE Publications 
1307 New York Avenue, NW, Suite 300 
Washington, DC 20005-4701 
Tel: 202/293-2450 

AACTE FaX: 202 / 457 " 8095 

1 ^ Web site: www.aacte.org 
Single copy: $10.00 AACTE members /nonmembers 




Copyright © 2000 by American Association of Colleges for Teacher 
Education. All rights reserved. Printed in the United States of America. 

ISBN No: 0-89333-179-1 




Assessing the Impacts of Professional Development Schools 



TABLE OF CONTENTS 



Editor's Preface v 



Acknowledgments viii 



Chapter 1 


The Challenges and Importance 
of PDS Impact Documentation 


1 


Chapter 2 


A Conceptual Framework 
for Assessing PDS Impacts 


3 


Chapter 3 


Designing and Implementing 
Impact Research: Suggested Steps 


9 


Chapter 4 


Tool Kit for Analyzing Impact Research Plans 
and Implementation 


13 


Chapter 5 


Profiles of Selected PDS Impact 
Research Efforts 


17 


Chapter 6 


Sample Protocols, Questionnaires, and 
Analytic Approaches 


25 


Chapter 7 


References and Additional Resources 


61 


About the , 


Authors 


68 



O 



5 



EDITOR’S PREFACE 



Some of the liveliest conversations taking place among educa- 
tors during the past decade have centered on professional develop- 
ment schools (PDSs). These conversations cover a lot of ground. There 
is talk of preservice teacher cohorts, site-based coursework, collabo- 
rative decision making, simultaneous renewal of schools and teacher 
education, new boundary-spanning roles for school and college fac- 
ulty, and school-based inquiry. However, it is not unusual for the easy 
flow of conversation to become halting, self-conscious, stiff, or list- 
less if someone outside of the fold, or even within, raises the ques- 
tion of documenting impact. The hapless individual who mentions 
the issue is apt to feel he or she has said something in bad taste. A 
certain skittishness often invades scholarly and informal discourse 
when the topic surfaces. 

The reluctance of professional development school implementers 
and advocates to come to grips with impact documentation can be 
traced to several understandable factors, which are outlined by the 
author of this work, Lee Teitel, in the introductory chapter. Never- 
theless, there is increasing pressure on those involved in PDS work 
to justify the considerable human and fiscal resources devoted to PDS 
initiatives. 

Pressure comes from legislators, state and district administrators, 
foundations, and higher education administrators who are asked to 
supply funds for PDS work. It comes from skeptical teacher educa- 
tors prodded to abandon established models of preservice prepara- 
tion and from weary teachers asked to add mentoring, inquiry, and 
management tasks to their classroom duties. It comes from cynical 
journalists, confused parents, curious researchers, and self-appointed 
guardians of taxpayer interests. Particularly pressing is the call for 
evidence that PDSs positively impact P-12 student outcomes, and 
other critical areas including new teacher placement, performance, 
and retention. 

If professional development schools are to prosper, or even sur- 
vive, the call for credible evidence of accomplishment cannot be dis- 
missed as irrelevant, misguided, or sinister. It is a legitimate demand 
that rests on solid ethical and practical ground. The present work 

O 

o 
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attempts to help PDS implementers who recognize the legitimacy of 
that demand overcome a significant obstacle to answering it. Shar- 
ing the work of those who have come to grips with the task and initi- 
ated activity to document PDS impacts on various stakeholders, this 
booklet helps to answer a frequently asked question, "How do we go 
about it?" 

PDS distinctiveness lies in its multidimensional mission. Profes- 
sional development schools are P-12 schools, some public and some 
private, developed and operated by collaborative partnerships, which 
include one or more schools, colleges, or departments of education 
and one or more schools or school districts. Some partnerships also 
include teachers unions and human service agencies as working part- 
ners. The Clinical Schools Clearinghouse (CSC), based at the Ameri- 
can Association of Colleges for Teacher Education ( AACTE), has docu- 
mented more than 1,035 P-12 schools designated as PDSs in the United 
States. 

The four-part mission of the PDS includes: (a) initial preparation 
of teachers and other school-based educators; (b) professional devel- 
opment of teachers and other school-based educators; (c) exemplary 
practice that supports, enhances, and improves P-12 student learn- 
ing; and (d) ongoing applied inquiry that supports student and edu- 
cator development. PDS partnerships are committed to equity in 
policy and practice for all learners, simultaneous renewal of schools 
and teacher education, and parity among partners. It is not the objec- 
tive of the PDS movement to have every school assume the multiple 
mission of the PDS; however, dissemination is a key function, and 
promising structures and practices are shared. PDSs are places de- 
signed to generate, test, and refine effective structures and practices 
that can be shared with other schools. 

This publication offers a conceptual framework for conducting 
impact documentation, a user-friendly checklist for assessing exist- 
ing documentation efforts or guiding those in the planning stages, 
and descriptions of existing impact documentation activity at a num- 
ber of sites. A key component is the collection of instruments, proto- 
cols, and analytical approaches developed by several partnerships. 

This booklet is designed to assist PDS planners and implementers 
seeking guidance in designing and carrying out impact documenta- 
tion. It can also be a useful resource for students, researchers, and 
policymakers interested in investigating PDS effectiveness. While no 
claims are made that the present work represents a comprehensive 
treatment of a very complex subject, it is offered as an aid to further 
the work of those who recognize the need for credible assessment and 
seek guidance in developing productive mechanisms and approaches. 
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Assessing the Impacts of Professional Development Schools is the sec- 
ond in a series of practice-oriented booklets on PDSs, The PDS Prac- 
tice Series, published by AACTE. The first booklet, Designing Profes- 
sional Development School Governance Structures, offered a practical, 
nuts-and-bolts guide to crafting institutional agreements to establish 
and manage PDS partnerships. Comments on this booklet and the 
series are welcome. 



Ismat Abdal-Haqq 
Series Editor 
May 1999 
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CHAPTER 1 

THE CHALLENGES AND IMPORTANCE 
OF PDS IMPACT DOCUMENTATION 



Introduction 

Credible, systematic documentation of the impacts of professional 
development schools (PDSs) is critical to the growth and sustenance 
of the partnerships themselves and of the PDS movement. Without 
good documentation of impacts on preservice and experienced edu- 
cators and on P-12 students, the professional development schools 
that have grown so fast in the last decade will wither away. Good 
impact documentation, carefully conceptualized and implemented, 
allows insiders to improve what they are doing and helps all stake- 
holders to assess whether the effort that goes into starting and sus- 
taining a professional development school is worth it. Yet, credible 
and systematic documentation of PDS impacts is hard to implement 
for several reasons: 

• The definition of professional development schools is not uni- 
versally agreed upon, so an evaluation might be tainted by look- 
ing for impacts within a setting that is not really a PDS. 

• It may be too soon to measure impacts for these partnerships; 
they represent long-term systemic changes that should not be 
measured before the changes are in place. Furthermore, relation- 
ships, which undergird collaborative partnerships, are fragile and 
can easily become damaged by a premature evaluation or one 
that is not sensitive to the need to nurture the relationship. 

• It is difficult to establish comparison or control groups; thus, 
self-selection or program selection, or some other confounding 
factor, may produce preservice students in PDS programs who 
may be different from those in more traditional campus-based 
programs. 

• There are different perceptions of what outcomes matter and how 
to measure them, especially among the different stakeholders. 

• Participants are too busy making the partnership happen to docu- 
ment their work; yet, reliance on outside evaluators who are of- 
ten brought in well after the start up of the PDS may lead to a 
mismatch between the goals of the program and the direction of 
the evaluation. 




10 




Not surprisingly, given these challenges, there are not many cred- 
ible and systematic assessments of PDS impacts available. Several 
recent literature reviews bemoan the paucity of high-quality evalua- 
tions, noting that many of the studies are limited by unclear descrip- 
tions of the methodologies and sometimes confounded by difficul- 
ties over publishing what may be seen as judgmental accounts of 
ongoing processes and relationships. 

Most of the published studies of PDSs focus on start up stories, 
roles and relationship changes, and teacher and student teacher atti- 
tudes, expectations, and satisfaction levels with little concrete evi- 
dence about how the quality of learning for P-12 students, preservice 
teachers, and experienced educators has been affected by the changes. 
Furthermore, credibility of many studies is undercut by exclusive 
reliance on self-report survey data. Most of the impact documenta- 
tion focuses on preservice teachers, with less attention paid to im- 
pacts on experienced teachers, and even less on administrators or 
faculty at the college, and still less on P-12 students (Abdal-Haqq, 
1998; Book, 1996; Teitel, 1998). 

The balance of this booklet is divided into two sections. The first 
part, Chapters 2 through 4, focuses on how to plan for and imple- 
ment PDS documentation work. It features a framework for thinking 
conceptually about PDS impact assessment and some recommended 
steps for implementation. It concludes with a self-assessment check- 
list designed to be used by readers to stimulate discussion and plan- 
ning of PDS research. The second section focuses on resources. Chap- 
ter 5 includes brief descriptions of several PDS research initiatives 
currently underway in this country. Chapter 6 consists of sample pro- 
tocols, questionnaires, and analytic approaches. The final chapter 
contains references and other resources. 
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CHAPTER 2 

A CONCEPTUAL FRAMEWORK 
FOR ASSESSING PDS IMPACTS 



The biggest and, ultimately, most important questions asked in 
any research on professional development schools concern impacts — 
impacts that produce improved student learning outcomes; improved 
preparation of preservice teachers, administrators, and other educa- 
tors; and improved, continuing professional development and learn- 
ing for all school- and university-based adults who work in the part- 
nership. 

In general, neither PDS participants nor researchers have at 
present a clear enough idea of what connections exist (if any) be- 
tween the structural change of setting up a PDS and desired out- 
comes. As Kimball, Swap, LaRosa, and Howick (1995), warn in writ- 
ing about the National Network for Educational Renewal partner 
schools: 

The means to effective partnership can easily become ends 
in themselves. For example, the energy for change in schools 
may become focused only on improving working conditions 
for teachers, establishing more collaborative decision-mak- 
ing structures, or creating more flexible schedules, all of 
which can be means to the end of the learning but should 
not be ends in themselves. Administrative practice can change 
without passing the advantage to the classroom, (p. 24) 

Writing about other forms of restructuring, Newmann and 
Wehlage (1995) use a series of concentric circles, with student learn- 
ing as the target in the center, and note that student learning is af- 
fected only where there are changes in the circle around it, which 
they label "authentic pedagogy." TTiis circle, authentic pedagogy, is in 
turn supported by improvements in the school's organizational capac- 
ity and its external supports, represented by the outer circles. They 
note that school restructuring without impacts on pedagogy will not 
change student learning. Slavin (1996, p. 4), reviewing a range of popu- 
lar reform or restructuring efforts, notes, "These reform proposals ig- 
nore a basic truth. Student achievement cannot change unless America's 
teachers use markedly more effective instruction methods." 



3 



O 



Following the same logic and working backward from the three 
goals listed at the beginning of this chapter, professional develop- 
ment schools need to demonstrate improved approaches to teaching 
and learning and leadership for three major stakeholder groups (i.e., 
preservice educators, inservice educators, and P-12 students). Con- 
tinuing to work backward, these sites must create contexts for the 
structural, organizational, and cultural changes that will encourage 
and support those improved approaches. A conceptual model, which 
incorporates these outcomes and the changes that will support them, 
is presented in Table 1. 



Table 1 
Concept Map 
PDS Impact Assessment 



Column 1 

Organizational 

Innovation: 

Partnership 

Development 


Column 2 

Adaptations in 
Roles, Structures, 
and Culture 


Column 3 

Best Practice 
in Teaching, 
Learning, and 
Leadership 


Column 4 

Desired Outcomes 
Improved Learning 
for: 


School 


Changes in 
governance, 
decision making, 
leadership; 


Classroom 
approaches 
"teaching for 
understanding," 
constructivism; 


All Students 


PDS 


use of time, roles, 
and rewards; 


different 
expectations for 
teacher 
professional 
development 


Experienced 
teachers and other 
education personnel 


Teacher 

Education 

Program 

(Also links to 
unions, parents, 
communities, 
districts, arts and 
sciences faculty) 


views of 
collaborative 
relationships, 
expectations 


different 
approaches to 
preservice 
teaching, 
fieldwork 


Preservice teachers 
and other education 
personnel 
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Using the PDS Impact Assessment Conceptual Map 

The conceptual map is drawn in a linear fashion. Changes pro- 
duced by setting up professional development schools are on the 
extreme left and lead to the desired outcomes which appear on the 
extreme right. In real life, of course, the process is much more recur- 
sive. The changes in Column 3 and in Column 2 often reach back and 
influence factors in Column 1 and so forth. Impact research cuts across 
the columns and should inform and connect the different pieces, in 
ways that are useful for both internal and external audiences. In the 
following paragraphs, each column is explained briefly along with 
suggested data sources. 

Column 1: Organizational Innovation: Partnership Development 

The organizational changes reported under this heading can be 
thought of quite broadly. For example, each of the structural entities 
"school" and "university" is surrounded by a larger community, each 
with its own (sometimes overlapping) sets of stakeholders and par- 
ticipants. Also, there are a number of subsections within each organi- 
zation — like arts and sciences faculties as well as teacher education 
programs, in a university; or parents, school boards, district office, 
for the school. PDSs cause new relationships to form between and 
among these different entities, often involving other community 
groups, agencies, and unions. Examples of data to be collected to 
document these organizational changes include minutes of meetings, 
collaborative agreements, histories, calendars of partnership events, 
surveys of stakeholders, press clippings and copies of newsletters, 
and annual progress reports (adapted from Sirotnik, [1988]). 

Column 2: Adaptations in Roles, Structure, and Culture 

To demonstrate that these partnerships have an impact beyond 
mere structural rearrangements, there needs to be evidence of other 
kinds of changes in the culture, governance structure, rewards sys- 
tem, use of time, human resource policies, and so forth — within all 
participating institutions. This includes development of new kinds 
of roles that cross over institutional boundaries, blur the distinctions 
between school faculty and college faculty, and change the nature of 
leadership and decision making in both organizations and cultures. 
These roles contribute to the development of new relationships and 
constituencies within the broader community as part of this process. 
These kinds of changes are necessary but ultimately, are not suffi- 
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cient to produce an impact on learning outcomes; they simply pro- 
vide the needed supports for changes in approaches to teaching and 
learning. 

Types of supporting data to collect include job descriptions and 
evidence of jointly conducted searches for personnel in new or modi- 
fied jobs at the intersection of the partner institutions; evidence of 
boundary-crossing capacities; changes in reward structure (e.g., pro- 
motion and tenure language) within the college and the school; oper- 
ating budget; and schedules that show how school and college time 
are used, as well as journal entries or reflections of participants. 

Column 3: Best Practice in Teaching, Learning, and Leadership 

Column 3 represents the next crucial link in assessing any types 
of impacts on preservice teachers, experienced educators, or P-12 stu- 
dents listed in Column 4 . Conclusions about PDS impacts on P-12 
students, for instance, only make sense if the students are experienc- 
ing better teaching and learning experiences in the PDS. Column 3 
documents the development of best practice models of teaching, learn- 
ing, and leadership — for children as well for as the initial and con- 
tinuing professional development of adults. 

Changes in the belief structure— the philosophy that underlies 
the teaching and learning and leadership practices — are as impor- 
tant as the actual changes in classroom practices. These changes could 
have implications for school organization (e.g., tracking vs. de-track- 
ing policies), as well as for numerous other innovations that may 
coincide with development of these partnerships (e.g., teaching for 
understanding or use of constructivist approaches). The roles and 
support for experienced educators, including administrators, in the 
school and college would be changing within this context of changes 
in practice. Leadership roles would be different, with co-teaching and 
research roles being developed in a more collaborative way, for ex- 
ample. In preservice programs, both the field and classroom experi- 
ences for preservice teachers would need to incorporate adoption and 
modeling of best practice approaches to instruction. Data to be col- 
lected include syllabi and course sequencing (for P-12 students as 
well as preservice teachers), evidence of effective strategies for work- 
ing with diverse students, professional development opportunities 
and plans for experienced educators, evidence of the beliefs underly- 
ing the practices, and documentation of what experienced teachers 
do with out-of-classroom time made possible by the presence of in- 
terns. 
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Column 4: Desired Learning Outcomes 

The findings in this column are ultimately the most important 
but make more sense when linked to the first three. Data need to be 
collected in ways that work for the various stakeholders and that are 
credible for both formative and summative decisions and collected 
on all three outcome categories (i.e., impacts on P-12 students, 
preservice educators, and inservice educators). 

Data should include multiple measures. For example, evidence 
to support the outcome of improved learning for preservice teachers 
might include perspectives of hiring principals, classroom observa- 
tions of graduating teachers, and P-12 student test scores (Stallings, 
1991). Additional data sources might be questionnaires on preservice 
teacher preparedness — drawing on views of preservice teachers, 
school-based mentors, and college faculty (Loving, Wiseman, Cooner, 
Sterbin, & Seidel, 1997) — along with archival data on graduates (i.e., 
where they applied to work, where they were hired, and the results 
of follow-up assessments of their preparedness and teaching skill over 
time). 



Using the Conceptual Framework 

The framework provides a logical, user-friendly way to link struc- 
tural, cultural, and governance changes to new approaches to teach- 
ing and learning, and ultimately to desired outcomes. It can also help 
identify potential gaps within a large study or assist in putting smaller 
studies in context. 

An earlier version of the framework was used at an American 
Educational Research Association (AERA) presentation that analyzed 
results of the large-scale, multisite study of the National Education 
Association's (NEA) Teacher Education Initiative (TEI). Among the 
dozen or so findings reported by Loving et al. (1997) from this study, 
there were 10 Column 2 (structural/ organizational /cultural) changes 
identified, five Column 3 (approaches to teaching, learning, and lead- 
ership), and only one Column 4 (impact on learning) outcome. The 
use of the framework highlighted the stage of the research and sug- 
gested additional data sources to the researchers. 

Randy Flora, who coordinates partner school activities at Miami 
University (OH) and was a participant in that meeting session, ex- 
pressed his concern about the usual simplistic demands for immedi- 
ate change of student performance on proficiency / achievement tests 
as the sole criterion of value. He noted that reliance on one set of 
measures and insistence that change be immediately observable on 
those measures would exclude other evidence of student learning 
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and would not recognize changes in environment, materials, teacher 
and other adult behavior, student health and safety, and other evi- 
dence that are often necessary conditions for change in educational 
results. Drawing on his recollection of legal terminology, Flora sug- 
gested that the conceptual framework presented provided a "pre- 
ponderance of evidence approach" — where the range of evidence 
displayed would suggest that the changes being instituted are im- 
pacting teaching and learning processes in ways that are consistent 
with research and literature on best practice, and show the potential 
for improved results for students over the long term. 

In addition to suggesting a logical link between PDS changes and 
potential impacts, the conceptual model can be used to provide a 
common framework to link smaller studies, which might be done at 
individual partnership sites. For instance, it is being used in Massa- 
chusetts to promote cross-site linkages and integration in develop- 
ing the "big picture" of the impacts that several different PDSs may 
be having at the local level (see. Chapter 5, item 5f and Chapter 6, 
Massachusetts Consortium for Initial Professional Development of 
Teachers). 
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CHAPTER 3 

DESIGNING AND IMPLEMENTING 
IMPACT RESEARCH: SUGGESTED STEPS 



What follows are some recommended steps for planning and 
conducting PDS impact research. The plan is oriented toward inter- 
nal documentation efforts — toward those partnerships that choose 
to plan and conduct the bulk of their documentation by themselves. 
It can certainly be adapted to those partnerships relying more heavily 
on external evaluators, but it does incorporate an important goal- 
setting and planning process that needs to be conducted by and for 
the stakeholders. 

The approach is designed to dovetail with the conceptual frame- 
work outlined in Chapter 2 but should serve as a useful checklist and 
guide for any PDS impact research. This approach is currently being 
used by a multisite research effort in Massachusetts, involving four 
partnership sites and the state department of education. It is described 
more fully in Chapter 5 (item 5f), and sample documents and data 
collection instruments used in Massachusetts are included in Chap- 
ter 6 (Artifacts A-H). The broad recommended steps are: 

1 . Focus on outcomes: What question(s) about impacts 
need to be asked? 

The first step is to identify what impact areas matter to your part- 
nership. This may be done initially by brainstorming the important 
questions with the research team but also should involve, early and 
often, all the various PDS stakeholders. For each area of interest (e.g., 
impact on preservice teachers, on P-12 students, on experienced edu- 
cators) develop a list of outcome attributes, the presence (or absence) 
of which would help determine success of the PDS. Work with the 
research team and stakeholders to focus the list on the core outcomes 
your program seeks. In keeping with the spirit and purposes of PDSs, 
it is critical that these conversations involve all stakeholders and that 
there be parity among university, school, and community participants 
at every step of the process. (See Chapter 6, Artifact A for an example 
of what a group of PDS participants from schools and colleges in 
Massachusetts identified as outcomes for preservice educator prepa- 
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ration. See also Chapter 6, Artifact I for core questions developed for 
organizing data collection from high school students in the Benedum 
Collaborative in West Virginia.) 



2. Determine the sources of information that can help 
answer the question(s). 

Think about the information you want to collect, selecting from 
a broad array of data sources, always keeping in mind what will be 
seen as credible by various stakeholders inside and outside the PDS. 
Preferred data sources are not necessarily immutable. For instance, 
individuals who consider standardized test scores as the only mea- 
sure of student achievement may be swayed with other forms of cred- 
ible evidence. (See, Chapter 5, item 5g, for a description of the vari- 
ous approaches taken by the Washington PDS/San Jose State 
University collaborative and sample instruments developed by the 
partnership, Chapter 6, Artifacts M and N.) 

Data points should go beyond exclusive reliance on self-report 
surveys but should provide multiple perspectives. For example, see 
the data grid used by Massachusetts researchers (Chapter 6, Artifact 
C) to map outcomes for preservice teachers with potential data 
sources. Before finalizing the data collection plan, it is important to 
get buy in from stakeholders on whether the data sources are appro- 
priate, comprehensive, and credible. 

A second important consideration is making sure that the data 
collection plan is feasible, given the available time and other resources 
within the partnership. These are less important issues if the partner- 
ship is well funded or has a separate funding stream for an external 
evaluation. Most PDS research efforts, however, are underfunded, 
and many are being implemented by busy people with many other 
demands on their time. Therefore, it makes sense to think about how 
to work most efficiently. 

Think imaginatively about archival data — information that al- 
ready is collected in your partnership — test scores, grades, and at- 
tendance rates for P-12 students; numbers of teacher education gradu- 
ates placed in jobs, etc. (See Chapter 5, item 5h for a description of a 
PDS research model, which incorporates existing data sources devel- 
oped by the Old Dominion University /Norfolk (VA) Public Schools 
partnership.) 

When there is information that you need to gather specifically 
for this study, try to weave it into the natural and educationally ben- 
eficial processes of your partnership. For example, if you want to get 
teacher education student attitudes on a variety of topics, consider 
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having them write "educational platforms" articulating their views. 
If this is done early in their program and again near program comple- 
tion, it not only provides good comparative data over time, but it can 
promote reflection that helps students develop and prepare for job 
interviews. 

Finally, make sure, in all your planning, to identify who will do 
the collecting, sorting, and analyzing of the data. This is a significant 
amount of work, which requires release time or additional compen- 
sation. Additionally, if the documentation effort is done internally, 
individuals may need some training or skill building around research 
techniques, especially if the research team includes, as it should, 
school-based personnel or community members who may be less 
likely to have research experience. (See Chapter 5, item 5e, for a de- 
scription of program support for site-based research within the West 
Alabama Learning Coalition, and Chapter 6, Artifact P for a table 
developed by the Coalition to summarize the outcome documenta- 
tion phase of its evaluation design.) 

3. Collect the data and conduct the analysis. 

The research work should always be grounded in some sort of 
conceptual map or framework similar in purpose to the one outlined 
in Chapter 2. Avoid, for example, looking at any of the outcomes in 
isolation from the changes that might be leading to it. For instance, if 
the data show growth and professional development of experienced 
faculty (See Table 1, a Column 4 change), map backwards to look at 
what activities, approaches, attitudes, and structures contributed to 
those gains (See Table 1, Column 3). If you are looking at developing 
positive attitudes of teacher education candidates toward urban 
schools, look in their preparation program for experiences that would 
lead preservice teachers to develop these attitudes and document the 
connection. 

Similarly, keep your eyes on the overall prize. Since one of the 
important underlying impact issues concerns the effects that any of 
these approaches for teacher education students ultimately have on 
kids, weave that question into your interviews and observations, and 
look specifically for impacts on the students in the school. If you are 
using a conceptual framework, it allows you to link items from dif- 
ferent data sources. (See the Massachusetts Consortium for Initial 
Teacher Professional Development of Teachers, Chapter 6, Artifacts 
D, E, and H, for a sample questionnaire and interview and observa- 
tion protocols used to gather outcome data on preservice educators 
from multiple sources.) 
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4. Continually engage stakeholders with the emerging data. 

"Presentation" should be the continual and iterative engagement 
of a variety of stakeholders with the findings — not a one-shot formal 
presentation, but ongoing connections to reach the people who care 
(or should care) about what you found. If you have already received 
stakeholder consensus that the outcomes you are looking at matter, 
and that the information you are collecting is credible, the key work 
becomes sharing, engaging, and analyzing. Think about the best way, 
when and where, to reach each stakeholder group. 

The sharing of research findings should be done in a way that 
maximizes impact on decision making. Avoid letting policymakers 
off the hook by allowing them to say, "you have to find the impacts." 
Instead, work with them to look at the impacts together. (See the 
Massachusetts Consortium for Initial Teacher Preparation Develop- 
ment, Chapter 6, Artifact G, for a sample of specific audiences the 
Massachusetts group felt would be important to reach with its find- 
ings and their planned methods of "presentation"). 

5. Refocus, refine, and continue. 

The ongoing impact documentation process should be inter- 
twined with presentation, analysis, and decision-making processes. 
It is important to plan ahead and think about future cycles of docu- 
mentation, looking, where possible, for baseline data. (See the Mas- 
sachusetts Consortium for Initial Teacher Preparation Development, 
Chapter 6, for a depiction of the multidimensional, long-term impact 
research efforts in which one PDS partnership is engaged, Artifact F, 
and samples of protocols and instruments used by Massachusetts 
Consortium members). 
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CHAPTER 4 

TOOL KIT FOR ANALYZING IMPACT 
RESEARCH PLANS AND IMPLEMENTATION 



The following self-assessment checklist may be used by PDS part- 
nerships in assessing the implementation of their impact research. It 
may also be a useful lens through which to view plans for designing 
and implementing PDS impact research. 

Using a scale ofl to 5, assess how well your impact research plan ad- 
dresses each of the steps outlined below, with 1 as poor, 5 as excellent, and 
NA for items that seem not applicable in your context . Use the " Planned " 
category if this aspect is already planned but not yet at a point where it can 
be meaningfully assessed. Use the "Indicators" section below each self-as- 
sessment rating scale to list afexv indicators that give evidence to support 
your assessment. If you indicated that an item was being planned, give some 
indication of the stage of planning or level of progress. 

Self- Assessment Checklist 

1. The impact research questions are clear and focused. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

2. Among PDS stakeholders there is widespread ownership and 
sense of the importance of the research question(s). 

Assessment of your partnership on this item: 



1 2 3 

(Poor) 



4 5 NA PLANNED 

(Excellent) 



Indicators: 



22 



13 



3. The design and implementation of the documentation effort in- 
volves all school and college partners in ways that reflect collabo- 
ration and parity. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor (Excellent) 

Indicators: 

4. There is a broad range of data sources being utilized to provide 
multiple measures. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

5. The data sources being used are seen as credible by all stake- 
holders. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

6. The documentation plans specify who will do the data collec- 
tion, analysis, and presentation, and those individuals have the 
needed skills and the time or other compensation to properly do 
the work. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 
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7. The findings are grounded in some sort of conceptual map or 
framework and are not just identified outcomes or processes iso- 
lated from one another. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

8. The "presentation" stage (whether it has happened yet or not) 
includes a continual and iterative engagement of stakeholders with 
the findings. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

9. There is a clear plan for multiyear, ongoing documentation. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 

10. There are clearly delineated resources to provide time or other 
compensation to those engaged in the documentation. 

Assessment of your partnership on this item: 

1 2 3 4 5 NA PLANNED 

(Poor) (Excellent) 

Indicators: 
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CHAPTER 5 

PROFILES OF SELECTED 

PDS IMPACT RESEARCH EFFORTS 



What follows is a sampling of some of the more thoughtful and 
innovative PDS impacts research studies going on around the coun- 
try. Where possible, citations (see reference list) of papers, reports, 
articles, or conference presentations are included, along with contact 
information for a lead researcher on the project. 

5a. The Benedum Collaborative involves the University of West 
Virginia and schools in four counties. Saab, Steel, and Shive (1997) 
focus on classroom impacts in the Benedum Collaborative, describ- 
ing a multipronged approach to data collection, including a teacher 
survey, teacher interviews, focus group interviews with students, and 
phone calls with a random sampling of parents. They include samples 
of student work and parent responses and describe how the data are 
being used to fine-tune the restructuring experience. Webb-Dempsey 
(1997) describes, in thoughtful detail, the collaborative process 
Benedum uses to gather and analyze data in a way that respects the 
needs and concerns of the participants while retaining a strong focus 
on getting credible data about impacts on experienced teachers, stu- 
dent teachers, P-12 students, and the schools themselves. 

Benedum draws on interviews with 400 students, surveys of 3,000 
students, evaluation data collected by the state department of educa- 
tion, as well as standardized tests (For an update on findings in these 
areas, see Teacher Education Research Group, 1999). Webb-Dempsey r s 
chapter, devoted to the work of the Benedum Collaborative, includes 
core questions used in semi-structured, open-ended focus group in- 
terviews with elementary and secondary students (See Chapter 6, 
Artifact I, for the questions.). The paper by the Teacher Education 
Research Group (1999, pp. 12-27) provides items and descriptions 
for several scales and sub-scales of the surveys utilized in the study. 
Three of the surveys are included in Chapter 6, Artifacts J, K, and L. 



For more information, contact: 

Van Dempsey, Teacher Education Research Group 

Benedum Collaborative 

College of Human Resources and Education 

West Virginia University 

Morgantown, WV 26506 
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5b. The Colorado Partnership for Educational Renewal is en- 
gaged in a complex assessment project involving six universities and 
12 school districts. The " empowerment evaluation" approach they 
use is consciously designed to assess what impacts are taking place 
at the local partnership level, as well as at the statewide Colorado 
Partnership level (Kozleski et al., 1997). Research teams have used a 
"portraiture" process to develop in-depth case studies of four of the 
partnerships, drawing on data from: 

• an activities log detailing all activities associated with the part- 
nership over the year. 

• focus group interviews of key stakeholders, including teach- 
ers, students, and administrators at school and university lev- 
els, as well as parents. Interviews were conducted by teams 
from another partnership site who were trained in interview 
techniques. 

• surveys, to get a broad base of information from a range of 
stakeholders. 

• classroom observations, using the Stallings Observation Sys- 
tem, which were conducted to compare classroom interactions 
with and without preservice teachers in the room. Other quali- 
tative data were also collected. 

At the Colorado Partnership level, these data from the four sites 
have been combined to create a snapshot of what is happening at the 
partner school sites, to look for evidence of implementation of the 
four functions of professional development schools and other bench- 
marks, and to assess the Colorado Partnership's impact on simulta- 
neous renewal efforts (Colorado Partnership for Educational Renewal, 
1997; see also Boland, Chandler, Kozleski, and Sueltz, 1999). 

For more information, contact: 

Cori Mantle-Bromley 

Colorado State University 

School of Education 

Fort Collins, CO 80523-1588 
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5c. The Teacher Education Initiative (TEI), supported by the 
National Education Association (NEA), is in the middle of a five- 
year, longitudinal "Seven-Site Replication Study of Teacher Prepara- 
tion/' which focuses on the impacts of TEI partnerships from the per- 
spectives of multiple stakeholders: P-12 students, parents, mentor 
teachers, preservice teachers, school administrators, superintendents, 
university faculty and administrators, and local teacher associations. 

While TEI encourages local, context-specific adaptation of the 
research design, the study relies on several common surveys, inter- 
view and case study protocols, and archival data sources to facilitate 
cross-site comparison. Data from interviews, focus groups, partici- 
pant narratives, and surveys (including comparative longitudinal 
career data on TEI preservice teachers and traditional program 
preservice teachers), as well as archival data, have been gathered since 
1995. Data collection instruments are included in the evaluation 
manual developed by Ross and Howze (1995) for the study. 

The study focuses on participant perceptions of changes in teacher 
preparation and how well the nine TEI "Guiding Principles," which 
undergird TEI partnership work, are being implemented. The prin- 
ciples are organized around the following topics: partnerships, lead- 
ership roles, evaluation and dissemination, professional preparation 
and development, external systemic change, internal systemic change, 
technology, equity and diversity, and teaching and learning. Since 
the study's inception, there have appeared a number of related ar- 
ticles and papers, which discuss the overall project (Pines, Seidel, & 
DiTrani, 1998), as well as activity and outcomes reported by the sites 
(Carnes & Boutte, 1998; Harriman, 1998; Loving et al., 1997; Wiseman, 
Ross, & Rakow, 1998). 

For more information, contact: 

Sylvia Seidel 

Teacher Education Initiative 
National Education Association 
1201 16 th Street NW, Suite 310 
Washington, DC 20036 
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5d. The Houston Consortium for Professional Development and 
Technology Centers is a member of the Texas statewide network of 
Centers for Professional Development of Teachers (CPDT), originally 
known as Centers for Professional Development and Technology, es- 
tablished with state legislature appropriations in 1991 and 1992. It is 
an urban consortium of four universities, three school districts, and 
two intermediate school agencies. In 1992, the consortium began the 
process of designing and implementing a coordinated teacher prepa- 
ration program specifically for prospective urban teachers. The pro- 
gram is based in professional development schools, involves signifi- 
cant use of telecommunications, and was implemented for all 
prospective teachers in the participating universities. 

Evaluation of consortium program impacts on teacher education 
students and P-12 students incorporated multiple data sources, in- 
cluding surveys, classroom observation, state-mandated teacher can- 
didate test scores, and state-mandated standardized criterion-refer- 
enced P-12 achievement tests. Among the reported results of 
evaluative studies, Houston, Hollis, Clay, Ligons, and Roff (1999) in- 
dicate: (a) 43% of PDS teachers believed that their involvement in the 
program had produced positive differences in their teaching; (b) ob- 
servations of teacher candidates and review of state certification test 
scores revealed that candidates taught differently and made higher 
scores than a comparison group, and (c) overall, achievement test 
scores of P-12 students increased after ‘their schools became PDSs. 
Impacts documented by the Houston Consortium are included among 
data reported by Macy, Macy, & Kjelgaard in a 1996 state-sponsored 
evaluation of the CPDT network. 

For more information, contact: 

W. Robert Houston 
Houston Consortium 

Texas Center for University School Partnerships 
University of Houston 
Houston, TX 77204-2162 

5e. The West Alabama Learning Coalition is a consortium of 
professional development schools, which includes five universities, 
two community colleges, and seven school districts. A distinguish- 
ing feature of the coalition, coordinated by the Truman Pierce Insti- 
tute at Auburn University, is the goal of community and economic 
development, as well as simultaneous renewal of K-12 and college 
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and university education. These multiple goals are reflected in the 
development and implementation of the Coalition's evaluation plan. 
Partners include a broad spectrum of community and social service 
agencies and businesses, as well as public school and higher educa- 
tion personnel. 

The evaluation design incorporates action guidelines committed 
to by each partnership within the Coalition, the purposes of the Coa- 
lition, Holmes Group principles, and the draft standards developed 
by the NCATE PDS Standards Project. Reed, Kochan, Ross, and 
Kunkel (1999) outline the thoughtful process involved in crafting the 
collaborative evaluation, which is intended to provide data useful to 
both individual sites and the Coalition as a whole, and articulate a 
set of eight underlying assumptions undergirding their evaluation 
design (See Chapter 6, Artifact O). 

Among the notable features of the group's work are venues for 
sharing, in ongoing fashion, partnership progress with parents and 
community members; a guide or manual developed to facilitate col- 
lection of comparable cross-site data for site portfolios; designated 
evaluation contacts for each site; and evaluation training sessions 
conducted at semi-annual Coalition meetings. 

The Coalition's evaluation system incorporates three phases of 
analysis, which reflect the three phases of PDS development articu- 
lated by the NCATE PDS Standards Project (Levine, 1998). Phase one 
focuses on structural considerations, phase two addresses process 
considerations, and phase three deals with identifying and describ- 
ing outcomes. See Chapter 6, Artifact P, for an example of the objec- 
tives, questions to guide inquiry, examples of methods and strate- 
gies, potential data sources, and suggested designations of 
responsibility for phase three. Formative data are currently being 
collected for the first two phases. Gathered data will be used to as- 
sess the achievements of each site and the Coalition at the end of the 
first four-year cycle. 

For more information, contact: 

Frances Kochan 
Truman Pierce Institute 
Auburn University 
Auburn University, AL 36849 
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5f. The Consortium for Initial Teacher Professional Develop- 
ment of Teachers uses federal (Eisenhower) monies and involves the 
Massachusetts State Department of Education, several statewide pro- 
fessional development organizations, and four school-college part- 
nerships in an effort to develop and nurture innovative approaches 
to teacher education. The Consortium has three goals: (a) developing 
model teacher preparation programs, (b) influencing state policy on 
teacher preparation based on the field-tested models (and aligning 
them with other reform initiatives), and (c) supporting a network to 
improve teacher professional development. Three of the four school- 
college partnerships in the Consortium have developed professional 
development schools as their "model" teacher preparation program, 
and a key element of the initiative has been documenting these im- 
pacts as a way to think about influencing policy. 

The three partnerships have been using the conceptual model 
and the impact documentation process steps outlined in the previ- 
ous chapters of this book. Their preliminary research has identified 
more than a half-dozen structural and governance changes and a 
dozen "Column 3" changes (see Table 1) in the experiences of 
preservice teachers. They are using a variety of measures to assess 
"Column 4" impacts (see Table 1), including, some of the protocols 
included in Chapter 6 (see Artifacts A through H). 

For more information, contact: 

Lee Teitel 

Graduate College of Education 
University of Massachusetts-Harbor Campus 
100 Morrissey Blvd. 

Boston, MA 02125 
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5g. The Washington PDS/San Jose State University partnership 
is conducting impact assessment that focuses on equity in student 
achievement. Incorporating an approach known as "image-based 
research/' the research team has worked on "telling the story" to 
multiple audiences through a variety of media, including videotape. 
Arguing that standardized achievement scores, presented alone, lead 
to bias in evaluating equitable gains in student achievement, the part- 
nership draws on data from a variety of sources, including reading 
tests, running records, portfolios, district accountability instruments, 
and student /teacher interviews. One of the 20 pilot sites for the PDS 
Standards Project of the National Council for Accreditation of Teacher 
Education (NCATE), the collaborative is incorporating and integrat- 
ing its data collection into the "Critical Attributes" framework of that 
study (Hollingsworth & Whitenack, 1999). 

The work of the partnership is very much driven by conscious 
attention to ensuring academic and life success for the largely Latino 
children at Washington PDS. The current focus is on bilingual lit- 
eracy, in Spanish and English; preparing preservice, beginning, and 
veteran teachers to successfully teach literacy in urban schools such 
as Washington; and developing strategies that recognize and con- 
front the sociopolitical context in which teaching and learning occur. 
Empowering students and parents and encouraging them to take 
ownership and responsibility for literacy success is a feature of cur- 
rent activity. The "Research Buddies" program creates teams consist- 
ing of partnership faculty /researchers, students, and their parents 
who collect data designed to help develop greater parent involve- 
ment and measure growth in family support for children's education 
(See Chapter 6, Artifacts M and N). 

For more information, contact: 

Sandra Hollingsworth and David Whitenack 
Washington PDS/San Jose State University 
College of Education 
One Washington Square 
San Jose, CA 95192 
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5h. The Old Dominion University/Noif oik Public Schools part- 
nership employs a collaboratively developed PDS research model, 
which is very heavily focused on tracking impacts on student achieve- 
ment and behavior (Bowers & Evans, 1999). The model includes four 
strands: (a) enhanced student academic achievement; (b) effects of 
Consistency Management and Cooperative Discipline on discipline 
referrals; (c) teachers' beliefs about their schools; and (d) restructured 
preparation of teachers, counselors, and administrators. Data are be- 
ing collected on student achievement and learning- re la ted behavior 
and performance. Data sources include standardized test scores, state 
assessment standards for SCDEs and K-12 schools, discipline refer- 
rals, attendance, and honor roll participation. A 1998 evaluation re- 
port indicated significant reductions in discipline referrals, improve- 
ments in communications skills, and more positive teacher 
perceptions of school climate. 

For more information, contact: 

Rebecca Bowers and Donna Evans 
Darden College of Education 
Old Dominion University 
Hampton Blvd. 

ISforfolk, VA 23529 
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CHAPTER 6 

SAMPLE PROTOCOLS, QUESTIONNAIRES, 
AND ANALYTIC APPROACHES 



This chapter includes sample artifacts drawn from the PDS re- 
search efforts described in Chapter 5. We include them for two rea- j 
sons: the first is to allow interested readers to get a more complete 
sense of the range of ideas and approaches to data gathering and 
analysis that these research teams are using. Accordingly, the arti- 
facts are organized under the heading of the research initiative that 
developed them. 

The second purpose is to share some concrete examples of how 
researchers are trying to gather data on PDS impacts. We thank the 
authors of these documents for their willingness to share them, and 
we hope readers find the samples useful in stimulating ideas for data 
collection and analysis in their own partnerships. Some of the ex- 
amples are draft documents, or works in progress; readers may wish 
to contact the research initiatives for more information. While the 
content of each artifact accurately reflects the original, some docu- 
ments have been reformatted to accommodate printing constraints. 



Massachusetts Consortium for Initial 
Professional Development of Teachers 

The Consortium developed, with input from teacher educators 
at participating schools and colleges, a list of criteria related to what 
improved preparation of teacher education candidates might look 
like. After refining this list through two iterations of review by a va- 
riety of stakeholders at each partnership, the Consortium 
brainstormed potential sources of data. As participants began to use 
one or more data sources, they shared interview protocols and other 
sources and jointly developed some more quantitative measure to 
complement the observational and qualitative instruments in place. 

Artifact A, Outcomes for Preservice Educator Preparation, be- 
came the basis for organizing much of the data collection. Artifact B, 
Data Sources, details the range of sources the Consortium tried to 
bring to bear on assessing these outcomes. Artifact C, Potential Sources 
of Data, presents a grid developed by the Consortium to maximize 
the potential data sources for each item. Consortium members used 
the outcomes to shape observation and interview protocols. For ex- 
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ample, note the manner in which questions put to mentor teachers 
about interns parallel the outcomes categories in Artifact D, Cooper- 
ating Teacher/Mentor Teacher Survey. Similarly when Paul A. Dever 
School /Lesley College faculty conducted follow-up observations and 
interviews of their graduates, they used the same categories but found 
that it was better to put the set of questions about practice first (See 
Artifact E, Cohort 1 Interview Protocol). Each partnership was asked 
to think about long-term local plans for documentation. Artifact F, 
University of Massachusetts, Amherst/ Springfield Public Schools, 
illustrates one such plan. In addition, the Consortium brainstormed 
potential audiences, local, state, and national, for its research find- 
ings (See Artifact G, Plans for Sharing Massachusetts Consortium 
Findings). 
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ARTIFACT A 

Massachusetts Consortium for Initial 
Professional Development of Teachers 
Outcomes for Preservice Educator Preparation: 
Criteria for Determining If Teacher Education 
Graduates Are Well Prepared 



A — Attitudes 

1- teaching and learning 

2- expectations of kids 

3- how kids learn best 

4- role of the teacher in accommodating diverse learners 

5- goals of education (e.g., selecting and sorting vs. teaching & 
learning) 

6- urban schools (demystifying, etc.) 

B — Professional relationships 

1- supporting a collegial and learning culture 

2- sharing practice through peer observation, mentoring, and 
discussion of teaching 

3- taking leadership roles (outside the classroom) 

4- understanding school culture 

5- comfortable in whole-school setting 

C — Teaching and learning practice 

1- subject matter expertise 

2- understanding motivation of kids 

3- pedagogical concepts and constructs 

4- repertoire of skills for classroom management, maintaining 
momentum, etc. that accommodate and respond to diverse 
learners 

5- constructivist and inquiry-based approach 

6- team teaching and team learning, etc. 

7- assessment of student learning 

D — Reflection and continuous professional development 

1- reflective practitioner (individually and collaboratively) 

2- resilient (maintain convictions and focus in face of adversity) 

3- continued learning about teaching, subject, and motivation 
of diverse learners 

4- able to self-assess teaching 
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ARTIFACT B 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Data Sources 

When the criteria list was accepted by all site stakeholders, we 
turned our attention to finding sources of data to help document what 
impacts the programs were having along these criteria. The sources 
had to be credible to stakeholders, manageable within the constraints 
of participants' time, and had to provide multiple perspectives on 
these criteria. The data sources are described below. 

Preservice Teacher Interviezvs 

The preservice teachers themselves can be a good source of in- 
formation, especially about their attitudes and views of their own 
knowledge and skills in each of the other areas. Their perspective 
can be tapped at several points early on in the program, mid-way 
into the first year of teaching, near the conclusion of the program, 
and after graduation. Springfield Public School /Univeristy of Mas- 
sachusetts at Amherst 180 Days Program did formal taped exit inter- 
views with each graduate at the end of the first year (July 1997). Al- 
though these interviews predated the development of the criteria 
outlined above, the transcripts were coded to find useful informa- 
tion pertaining to the common framework. Paul A. Dever School/ 
Lesley College used the criteria to design a follow-up interview and 
observation protocol for use with 1997 graduates mid-way through 
their first years of teaching. Worcester Public Schools/Clark Univer- 
sity used focus groups with their 1997 graduates, also tailoring ques- 
tions to the agreed on criteria categories. In addition, one of the Mas- 
sachusetts Department of Education (DOE) coordinators, with input 
and approval of the sites, developed a survey questionnaire for gradu- 
ates along the same lines, with some standardized questions that could 
be more readily quantified. 

Preservice Teacher Reflective Writing 

This proved to be another good, and relatively easy, way to get 
data during the PDS year. Dever/ Lesley designed a series of writing 
prompts that helped students reflect and focus on important areas of 
their preparation. These "freewrites" in seminars or as homework 
assignments were collected and provided important insights, espe- 
cially into how attitudes were shaped overtime. In addition, they 
served as helpful reflective prompts and class discussion triggers. 
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Preservice Teacher Portfolios 

This represents a good example of a cost-effective source of data, 
using something that would be taking place anyway. Because stu- 
dents are asked to draw on a wide range of experiences and docu- 
mentation, their portfolios become a good source of data on a variety 
of measures. Worcester/Clark has discussed incorporating the 
agreed-upon criteria into the guidelines for the portfolio. 

Pupil Interviews , Writing , or Focus Groups 

Pupils are an often untapped source of information that provides 
a valuable and unique perspective on teaching approaches and atti- 
tudes. Although not done systematically for all of the categories above, 
Springfield/UMass Amherst interviewed children about the impact of 
the Legacy Projects, an after-school community service-learning program. 

Cooperating Practitioner/ Administrator Observations 

Since these observations would be taking place anyway, partici- 
pants can collect data in a way that does not require added work. In 
fact, using the desired outcomes as a guide can help focus observa- 
tions, helping observers look for desired features listed as part of 
teaching and learning practice and making the observations more 
useful data sources. Data from observations can be collected through 
focused observation sheets or through individual or group interviews. 

University Faculty Observations 

The rationale for and approach to university faculty observations 
are comparable to those of the cooperating practitioner/administra- 
tor observations described above. 



Archival Data 

These data include information that is available without any ex- 
tra work. In Massachusetts, at the time of this writing, teacher educa- 
tion students were required to take state subject area competency 
tests, which can be one source of data on subject area mastery. Records 
kept on peer coaching or " rounds" can help document the "sharing 
practice" subcategory of Artifact A, and evidence of leadership roles 
outside of classrooms (e.g. Legacy Projects) can be used to document 
the leadership subcategory. 



Administrators & Teacher Observations of New Hires 

This follow-up work in the schools employing graduates is being 
done by a DOE coordinator who, again guided by the desired out- 
comes criteria, developed a survey instrument and interview protocol 



£ nple Protocols, Questionnaires, and Analytic Approaches 




37 



29 



ARTIFACT C 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Potential Sources of Data 



The grid below maps outcomes for preservice teachers with potential data sources. 
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ARTIFACT D 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Cooperating feacher/Mentor Teacher Survey 

We are gathering information about typical and immersion teacher prepara- 
tion programs, including the one with which your school has been associated. 
Your responses are completely confidential and will only be reported as an ag- 
gregate so that individual responses cannot be identified. 

Thank you for taking the time to complete this survey. 

Section 1 - The Teacher Preparation Program 

Note: "Typical student teaching program" refers to a half or full practicum (usu- 
ally 8-13 weeks) as offered by most colleges and universities. This leads to a pro- 
visional certificate with advanced standing. "Immersion teacher preparation 
program" refers to a year-long school-based program such as the 180 Days pro- 
gram. Completion of both the student teaching and clinical teaching portions of 
this program leads to standard certification. 

The program with which the student teacher/ intern working with me has 
been involved is 

an undergraduate program 

a graduate program 

a "typical" student teaching program 

an immersion teacher preparation program 

I have worked with this student teacher/ intern 

as a cooperating practitioner for the student teaching portion only 

as a cooperating practitioner for the clinical teaching portion only 

as a cooperating practitioner for both the student teaching and 

clinical teaching portions 
in an other capacity: (define) 

Please continue! 

Feel free to add comments whenever you'd like. 
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Please use the following scale to indicate the extent of your agreement or 
disagreement with each statement: 

l=strongly disagree 2=disagree 3=notsure 4=agree 5= strongly agree 



After completing this teaching preparation program, I feel that my student 
teacher/ intern is now well prepared to: 

Strongly Disagree Strongly Agree 



maintain a positive attitude toward teaching 1 

set and maintain high expectations for all students 1 

use a variety of teaching strategies 1 

meet the diverse needs of the students 1 

implement the MA Curriculum Frameworks 1 

be sensitive to ethnic and cultural differences 

among students 1 

include special education students 1 

support a collegial and professional school culture 1 

work cooperatively with colleagues 1 

take leadership roles outside the classroom 1 

understand the culture of a school 1 

be comfortable throughout the school 1 

(in various settings) 

teach knowledgeably as a subject(s) area expert 1 

motivate students effectively 1 

use different kinds and methods of teaching as needed 1 
manage classroom activities 1 

deal effectively with discipline problems 1 

use technology as a tool for teaching and learning 1 

use inquiry to improve teaching and learning 1 

teach as part of a teaching team 1 

use multiple forms of assessment to evaluate 

student learning 1 

interact with his/her students' parents 1 

in meaningful ways 

reflect upon his/ her practice 1 

be resilient ("bounce back") after a difficult time 1 

continue to develop professionally 1 



2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 

2 



3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 

3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 

3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 

3 4 5 
3 3 5 

3 4 5 
3 4 5 
3 4 5 



What do you think is the greatest strength of this teacher preparation program? 

How would you improve this teacher preparation program? (Please describe a 
specific aspect of the program you would change.) 

Next year I plan on teaching in a different school. 

I plan on leaving teaching after this year and doing something else. 

I plan on teaching professional development activities in the next two 

years. 

I plan to work with teacher preparation programs in the next two 

years. If yes, list activities. 
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Which of the following describes your future plans? 

I definitely plan to be teaching five years from now. 

I hope I will be teaching five years from now. 

I may be teaching five years from now unless I find something better 

to do. 

I definitely plan on not teaching five years from now. 

I plan to be involved in teacher preparation programs five years from 

now as a mentor/cooperating practitioner. 

I plan on teaching teacher preparation at the college level five years 

from now. 

BACKGROUND INFORMATION 

At what grade level are you currently teaching? 

K-primary grades 

upper elementary grades 

middle school 

high school 

If you teach in a specific curriculum area(s), please indicate it (them) below: 

What is your certification level? 

Provisional with advanced standing 

Standard 

In what area(s) are you certified? 

What is your age bracket? 

21-24 years old 

25-29 

30-34 

35-39 

40-45 

45-50 

50 and older 

What is your ethnic/racial group? 

African-American 

Cape Verdean 

Hispanic 

Other 



Asian 

Caucasian 

Native American 



What is your gender? 

male 

female 



Thank you so much for your time and effort! 



Mentor Teacher Survey-Ann Barone 6-8-98 
New Teacher Survey- M. Fonseca 11-26-97 
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ARTIFACT E 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Paul A. Dever School/Lesley College Program Documentation 
Cohort 1 Interview Protocol 
Fall, 1997 



Tell me about a teaching experience this year that went really well, 
probes: 

What was your teaching approach? 

Talk about other teaching strategies you've tried. 

How did you assess student learning? 

Do you have the same comfort level with other disciplines? 

Attitudes 

What were your expectations for the students in this lesson? 
probes: 

Generalize to the broader expectations for students. 

How do you think children learn in your classroom? 
probes: 

Is that your ideal? 

How would you change this? 

Describe your school. 

How is it different/ same from the Dever? 

What is the ethnicity of the faculty? 

(Reminder: Ask for demographics from the office.) 

Relationships 

Describe the culture among the teachers in your school, 
probes: 

How do teachers interact with one another and with administration? 

How are decisions made? 

Talk about their experience team teaching. 

Talk about their perceptions of how other teachers receive them. 

What roles outside of the classroom have you assumed? 

Reflective Practice/Professional Development 

What challenges have you encountered as a first-year teacher? 

What was your worst disaster? How did you cope with it? 

As they think ahead, what plans do they have to stay abreast professionally? 
Are they talking to other people about their teaching? Explain. 

(Reminder: Ask for journals and portfolios.) 



Practice 
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ARTIFACT F 

University of Massachusetts-Amherst/ 
Springfield Public Schools 
UMASS/Springfield Flow Chart 
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ARTIFACT G 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Plans for Sharing Massachusetts Consortium Findings 

In addition to presentations at local and national conferences, 
the Consortium participants have worked hard to share their find- 
ings with local and statewide stakeholders in impactive ways. In 
November 1998, as a group, along with other Consortium members, 
PDS faculty presented their impact findings to date, to a representa- 
tive of the Massachusetts Commissioner of Education. 

Presentations are also planned to a variety of audiences including: 

• members of the collaborative 

• other faculty /administrators in school or district 

♦ other faculty/ administrators in college or university 

♦ area college consortia (e.g. Worcester group, or Boston higher 
education partnership) 

♦ other interested groups at school or college (e.g., Carnegie Com- 
mittee in Springfield) 

♦ school committee 

♦ school site councils 

♦ unions 

♦ parents, other school level stakeholders 

♦ state legislators 

♦ state department of education 

• professional journals, organizations 

Strategies 

As the Consortium moves forward in sharing findings, some guid- 
ing strategies have been discussed. 

• Think about what information is credible for the audience. 

• Think of what forum to engage them (preferably early on) with 
your findings. 

• Develop different versions of report of data, tailored to differ- 
ent audiences and using different media: 

• Use existing ways to get information out — get on agendas of 
existing groups, use superintendent's TV show, newsletters, etc. 

• Generate interest — tease people with short summaries of in- 
teresting findings in places where you can invite responses. 

• Keep connected to decision makers, so what you do has im- 
pact. 




ARTIFACT H 

Massachusetts Consortium for 
Initial Professional Development of Teachers 
Clark Uni versify- Worcester Public Schools 
K-16 Collaborative 
First Year Teacher Study 

Please answer the following questions as completely and hon- 
estly as you can. The information that you provide will help us to 
understand and assess the impact of our teacher education programs. 
Although you responses will be used as data to report to funding 
agencies, neither your name nor identifying information will be used 
without your explicit permission. If you have any questions, please 
call either Tom Del Prete or Maureen Reddy. 

1) What are you expectations for the children in your class? In 
answering this question, you might want to consider the fol- 
lowing; your role in fulfilling these expectations, the role of the 
school in general, how the child's home and family impact your 
expectations, and what you hope the children will learn. 

2) What is your model of an ideal professional culture? Please 
describe the professional atmosphere in your school and think 
about what, if anything, you would change to make it better 
match your ideal. Consider things such as whether teachers 
work together and the impact of the school culture on teachers. 

3) Describe a teaching experience this year that was very success- 
ful. What went into making it a success? As you answer these 
questions, please think about what, in your opinion, are the 
criteria for successful teaching and learning. 

4) Describe something that you tried in your class that did not go 
as planned. How did you recover form it? How will this expe- 
rience impact your teaching in the future? 

5) If you have taken part in any professional development activi- 
ties or if you have any planned for this year, please describe 
those activities. 
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Benedum Collaborative 

More than 400 elementary and secondary school students in 
Benedum PDSs were interviewed individually and in 3-5 member 
focus groups in the initial stages of the PDS Impact Assessment Study. 
Artifact I provides the set of core questions used across all interviews 
to elicit students' perceptions about their learning opportunities and 
descriptions of their school experiences as they related to the 
collaborative's guiding PDS belief statements (Webb-Dempsey, 1997, 
pp. 277). Interview findings helped to generate student survey items 
employed later in the process. (See Teacher Education Research Group 
[1999]) for survey items, data analysis, and discussion of the study's 
design.) Three artifacts are included from this study: an elementary 
student survey (Artifact J); a high school student survey (Artifact K); 
and a School Climate Educator Survey (Artifact L), designed to as- 
sess perceptions and attitudes of teachers about their schools. 

ARTIFACT I 

Benedum Collaborative 
Core Questions: K-12 Student Interviews 

• How do you feel you learn best? How do you know that is 
how you learn best? Do you have opportunities that encour- 
age you to learn that way in school? 

• What do you like to leam about in school? What's the best way 
for you to learn about that? Do you have opportunities to leam 
that way in school? 

• What do you do that makes you feel successful? Do you have 
opportunities to feel successful in school? What happens in 
school to make others feel successful? 

• What kinds of decisions/ choices do you make about your learn- 
ing? What kinds of decisions /choices are you involved in at 
your school? 

• How are people in your school different from one another? Are 
students treated differently because of their differences, if so, 
how? Are people's differences respected? 

• Is there anything about your experiences at school that I haven't 
asked you about that you think I should know? 
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ARTIFACT J 

Benedum Collaborative 

Elementary Student Survey 



Age: Grade:. 



Gender: Male/Boy 

Female/Girl. 



Ethnicity: (check one) 

White, not Hispanic origin 

Hispanic, Chicano, Latino 

American Indian 



Black, African American 

Asian, Pacific Islander 

Other (describe) 



1. I learn best when: 



2. I learn best then because: 



3. My parents learn when: 



4. I know my teacher(s) are learning new things when: 



5. Things that happen in school that help me learn are: 



6. Things that happen in school that do not help me learn are: 



7. The best lesson a teacher ever taught me was: 



8. In school I get to make choices about: 



9. To me, a successful student is: 
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10. Check the answer that describes you best and complete the sentence 
I am a successful student, because: 



I am not a successful student, because: 



11. Check the answer that describes you best and complete the sentence: 
I am a good student. I know that I am a good student because: 



.1 am an average student. I know that I am an average student because: 



.1 am not a good student. I know that I am not a good student because: 



12. Check the answer that describes you best and complete the sentence: 
I think I can change the kind of student I am. I feel that way because: 



I do not think I can change the kind of student I am. I feel that way 

because: 

13. My teacher(s) think it is important to: 



14. In my school, it is important to: 



I know my teacher(s) think that is important because: 
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15. In my school, it is important to: 



I know that it is important because: 



16. Check the one that best fits and finish the sentence: 

I think people in this school treat each other with respect 

because: 

I do not think people in this school treat each other with respect 

because: 



17. Check the answer that best fits and finish the sentence: 

I like having WVU university students in my classroom 

because: 

I do not like having WVU university students in my classroom 

because: 



18. Having WVU students is (Check One) 

the same different from having my regular teacher. 

Describe how it is the same or different: 



19. 



What I like most about using the computer is: 



I A O I 
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20. Here is a list of many different things people can do to learn. Check the 
ones that you have done at school this year. Think about whether you liked 
doing these things or did not like doing these things. Draw a happy face or a 
sad face beside the ones you checked to show which ones you like doing and 
which ones you did not like doing. 

field trip 

reading a book 

watching a video 

listening to someone read a book 

computers 

writing 

role play 

working with partners 

working by myself 

working with groups 

guest speakers 

drawing 

listening to music 

making music 

dancing 

singing 

PE 

tests 

doing "real" things 

doing worksheets 

experiments 

presen tations 

guest performers 

show and tell or sharing 

working with students at other grades 

stations 

painting 

displays 

making collages 

21. Check the one that fits and finish the sentence: 

I am an author 

because: 



I am not an author 

because: 
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22. Think about different kinds of writing, like letters, notes, journals, lists, 
and other things. 

What kinds of writing do you like to do: 



23. Describe how you feel about writing: 



24. When I read outside school, I read: 



25. Check the ones that fit: 

I read to my parents 

I read to my brothers or sisters 

I read to myself 

I read to a friend 

I read to my teacher 

I read to other students 

I do not read to anyone 

Other people I read to are: 



Thank You for Your Participation! 
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ARTIFACT K 

Benedum Collaborative 
1997 East Fairmont Senior High School 
Student Survey 

This survey is intended to describe your perceptions of your experiences in 
your school. To help you think about those experiences we are asking two differ- 
ent kinds of questions. Some questions ask you to answer by circling the number 
of the response that best describes your experiences. Other questions are open- 
ended and ask you to write your responses in your own words. Your participa- 
tion is entirely voluntary. Your participation, or lack of participation, will not 
have consequences for you or have any effect on your academic status. This in- 
formation from this survey will be used to asses the kinds of experiences stu- 
dents in your school are having. 

Age: Grade Level: Gender: (check) Female Male 

Ethnicity: (check) White, not Hispanic origin Black, African American 

Hispanic, Chicano, Latino Asian, Pacific Islander 

American Indian Other (describe) 



Academic Status: (check) 

College Prep Vocational Prep Occupational Prep 

Please responsd to each of the following questions by circling the response 
that is most descriptive of your experiences at East Fairmont Senior High 



School. 

Never 


Rarely 


Sometimes 


Usually 


Always 


1 


2 


3 


4 


5 



1. My individual learning needs are provided for. 1 2 3 4 5 

2. I have opportunities to make the most of my abilities. 1 2 3 4 5 

3. I am encouraged to take responsibility for my own 1 2 3 4 5 

learning. 

4. Iam encouraged to share my ideas about things lam 1 2 3 4 5 

learning with others. 

5. I have a variety of learning experiences. 1 2 3 4 5 

6. Iam encouraged to think about how I learn. 1 2 3 4 5 

7. What I learn in school will be useful in my life outside 1 2 3 4 5 

school. 

8. Opportunities to learn are available at the school for 1 2 3 4 5 

my parents or guardians. 
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Never Rarely Sometimes Usually 

1 2 3 4 

9. My parents or guardians meet with teachers or others 1 



at the school to discuss my learning. 

10. My teachers learn new things that they use with 1 

students. 

11. Having WVU student teachers in my classes is a 1 

positive factor in my learning experiences. 

12. What I learn in school will help me in the future. 1 

13. I have opportunities to be successful in a variety of 1 
ways in school. 

14. Grades are only one indicator of my success. 1 

15. I am a successful student. 1 

16. My teachers expect me to be successful. 1 

17. My teachers help me to be successful. 1 

18. I expect myself to be successful. 1 

19. 1 think I can change the kind of student 1 am. 1 



20. I have opportunities in school to pursue my learning 1 
interests. 



Always 

5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 



21. My experiences as a student in this school make me 1 2 3 4 5 

feel good about myself. 

22. I am challenged to learn and grow in school. 1 2 3 4 5 

23. I have opportunities to make choices about activities 1 2 3 4 5 

in my classes. 

24. My parents or guardians are encouraged to be involved 1 2 3 4 5 

in important decisions at this school. 



25. I have opportunities to have input into major 
decisions that affect school life. 



1 2 3 4 5 



26. I am encouraged to share my ideas with teachers and 1 2 3 4 5 

others in this school. 

27. The community I come from is respected in this school 1 2 3 4 5 
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Never Rarely Sometimes Usually Always 

1 2 3 4 5 

28. The cultural and racial background of students is 1 2 3 4 5 

respected in this school. 

29. I am treated with respect by teachers in this school. 1 2 3 4 5 

30. I am treated with respect by other students in this 1 2 3 4 5 

school. 

31. I am treated with respect by the administrators, like 1 2 3 4 5 

the principal, in this school. 



Please describe how you learn best. 

What experiences do you have in school that encourage you to learn that 
way? 

32. My teachers use various activities in their teaching. 1 2 3 4 5 

33. I have teachers who use portfolio assessments. 1 2 3 4 5 

34. I have teachers who re-teach material when students 1 2 3 4 5 

need help. 

35. I have teachers who have used a Polaroid camera 1 2 3 4 5 

to assist with an activity. 



36. I have WVU student teachers/observers. 



1 2 3 4 5 



37. My WVU student teachers/observers assisted with 1 2 3 4 5 

my learning experiences. 

38. I have experiences in school that help me appreicate 1 2 3 4 5 

cultural differences. 



39. I have experiences in school that help me appreciate 1 2 3 4 5 

the special needs of other people. 

40. I have had access to computer technology in school. 1 2 3 4 5 

41. I have had opportunities to use computer technolgoy 1 2 3 4 5 

in the majority of my classes. 

42. I have had experiences in school that help me know 1 2 3 4 5 

about different kinds of workplaces. 



43. I have had opportunities to explore career clusters. 1 



44. I have had opportunities to explore different college 
majors. 



3 4 
3 4 
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Never Rarely Sometimes Usually 

1 2 3 4 

45. I believe that what I learn in school is connected to 1 
what I will do after graduation. 

46. I have had experiences that have helped me gain 1 

skills related to a college major. 

47. My school experiences will be/have been helpful in 1 
making a decision about what I will do after graduation. 

48. I have thought of dropping out. 1 

49. I feel safe at this school. 1 

50. I feel the discipline at this school is fair. 1 

51. Overall, I am satisfied with my school experiences. 1 



Always 

5 

2 3 4 5 

2 3 4 5 

2 3 4 5 

2 3 4 5 
2 3 4 5 
2 3 4 5 
2 3 4 5 



Describe your thoughts about the classes offered at East Fairmont Senior High 
School (what is taught, how teachers teach, etc.) Are there other things you 
would like to see offered or other ways you would like to see classes taught? 

Please describe what you think a succesful student is like, and why you think 
that. 

Describe what helps you to be successful. 

Please describe your plans after high school. 

Please list the kinds of school experiences and/or opportunities you have had 
related to preparing for a job or college. 

Describe how your school experiences and/or opportunities are preparing/ 
have prepared you to make decisions about your future. 

Describe how your school experiences and/or opportunities are preparing/ 
have prepared you to pursue your plans after high school. 

Please use this space to make comments or suggestions about either your 
experiences as a student at East Fairmont Senior High School or this survey. 
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ARTIFACT L 

Benedum Collaborative 

School Climate Educator Survey (SCES) 



Much has been written of late about the reform of schooling practices. 
In examining these issues it is important to account for the perceptions of 
practicing educators who are most likely to be affected by change. The pur- 
pose of this survey is to collect opinions, perceptions and attitudes of prac- 
ticing educators regarding the climate of their schools. Please respond to 
each item by circling the response that is most descriptive of your experi- 
ences in your school. We assure you that the information you provide will 
be treated confidentially. Please do not put any identifying marks on this 
questionnaire. 



Please put the name of your school here: 



Please use the following scale to respond to the survey items 

Never Rarely Sometimes Usually Always 

1 2 3 4 5 

1. The administration in my school encourages teachers 1 2 3 4 5 

to take risks. 

2. University faculty are involved in the development 1 2 3 4 5 

of my school's programs. 

3. At our school educational research is seen as 1 2 3 4 5 

hepful in informing long-range planning. 

4. Classrooms in my school provide opportunities for 1 2 3 4 5 

students to exercise choices in how they will leam. 

5. Collaborative planning is a regular practice in our 1 2 3 4 5 

school. 



6. My school has acquired additional money through 1 2 3 4 5 

grant-writing and other external sources. 

7. Faculty at my school have been guest lecturers in 1 2 3 4 5 

university/college courses. 



8. Guidelines for student behavior are developed 
without the input of classroom teachers. 



1 2 3 4 5 



nple Protocols, Questionnaires, and Analytic Approaches 

57s-; 



49 



Please use the following scale to respond to the survey items 
Never Rarely Sometimes Usually 

1 2 3 4 

9. The local school improvement council (LSIC) is an 12 
integral part of our school. 

10. In addition to regularly scheduled conferences, 1 2 

teachers in our school meet with parents to discuss 
student learning. 

11. The design of progress reports reflects my school's 1 2 

mission. 



12. Our school's local business partner is involved in 12 

our school activities. 

13. Teachers in my school participate in the decisions 1 2 

about teacher's assignments to duties outside the 
classroom. 

14. Teachers in my school make decisions about day- 1 2 

today practice based on the school's long-range 

strategic plan. 

15. The professional development activities available 1 2 

to teachers in my school are planned by central office 
administratiors. 

16. Our school uses non- traditional scheduling and 1 2 



grouping strategies. 

17. It is necessary for students to understand the reasons 1 2 

for instructional strategies in order for them to be 
successful. 

18. Facutly members have the opportunity to contribute 1 2 

to decision-making about how money is spent. 

19. Decisions about how money is spent are consistent 1 2 

with our school mission. 

20. Our school building is used by community groups 1 2 

when school is not in session. 

21. My school has good atendance at back-to-school 1 2 

nights and at other school sponsored community events. 



Always 

5 

3 4 5 
3 4 5 

3 4 5 
3 4 5 
3 4 5 

3 4 5 

3 4 5 

3 4 5 
3 4 5 

3 4 5 
3 4 5 
3 4 5 
3 4 5 
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Please use the following scale to respond to the survey items 

Never Rarely Sometimes Usually Always 

12 345 



22. In my school teachers use educational research to 1 2 3 4 5 

develop teaching strategies. 

23. University faculty are involved in the 1 2 3 4 5 

implementation of my school's programs. 

24. School and university faculty are partners in the 1 2 3 4 5 

preparation of teachers and other educational 

professionals. 

25. My school hosts preservice teachers in practicum 1 2 3 4 5 

placements. 

26. Teachers in my school evaluate the performance 1 2 3 4 5 

of student teachers. 

27. Faculty in my school attend conferences to learn 1 2 3 4 5 

new teaching strategies. 

28. My school involves parents in curriculum development. 1 2 3 4 5 



29. At my school teachers look forward to inservice events 1 2 3 4 5 

more as opportunities to socialize than as an opportunity 

for professional growth. 

30. Families of the children who attend my school 1 2 3 4 5 

view it as a positive learning environment. 

31. In my school teachers meet with students to discuss 1 2 3 4 5 

their learning. 

32. Families at our school attend regularly scheduled 1 2 3 4 5 

parent-teacher conferences. 

33. When faced with a mandate (state, county), our 1 2 3 4 5 

school takes a proactive stance on how it will be 

implemented. 

34. Faculty at my school use newsletters or phone 1 2 3 4 5 

calls to maintain school-family communication. 

35. Faculty at my school teach in university /college 1 2 3 4 5 

courses. 

36. Volunteers assist in my school. 1 2 3 4 5 
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Please use the following scale to respond to the survey items 

Never Rarely Sometimes Usually Always 

1 2 3 4 5 



37. Students in my school are active learners. 

38. Remedial programs and other support services are 
available at my school for students who are not 
experiencing success. 

39. Instructional groups at my school are based on 
student achievement. 

40. Administrators in my school are committed to 
enhancing their professional knowledge and skills. 

41. The curriculum at my school reflects sensitivity to 
many cultures. 

42. Students at my school expect to do well. 

43. Teachers at my school enjoy teaching. 

44. The principal at my school is accessbile to staff 
members who are experiencing difficulty. 

45. The PTA/PTO at my school is active. 

46. At my school conflicts among the faculty are 
openly discussed in order to encourage resolution. 

47. Faculty at my school have helped design 
university/college courses. 

48. Teachers in my school participate in developing 
the agenda for faculty meetings. 

49. Teachers in my school participate in the process 
of hiring new staff members. 

50. Parent volunteers participate in and assist with 
classroom instruction. 



1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 

1 2 3 4 5 



51. Teachers in my school participate in the 
professional evaluations of administrators. 

52. Families are enocuraged to be involved in 
important decisions at my school. 



1 2 3 4 5 
1 2 3 4 5 
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Please use the following scale to respond to the survey items 

Never Rarely Sometimes Usually Always 

1 2 3 4 5 

53. Teachers at my school collaborate about students 1 2 3 4 5 

and their learning needs. 

54. My school has good relationships with the 1 2 3 4 5 

community. 



Thank you for completing the survey. Please feel free to offer any 
comments or suggestions. 



Please answer the following questions about yourself so that we can better 
interpert survey responses. Thank You. 

How many years have you taught? 

How many year have you taught at this school? 

Describe your role in the school, inculding the grades and subjects you teach. 
(Ex. Chapter 1 Reading (K-6), Math (7&8), Itinerant Art (1-6), Assistant Prin- 
cipal) 

What is the highest degree you currently hold? 

Bachelors Masters Masters +45 Doctorate 

Are you currently pursing any degree or certification program? 

No Yes: Please specify 



o 

FRir 
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Washington PDS/San Jose State University 

Washington PDS/San Jose State University (SJSU) has taken an 
innovative approach to developing multiple assessments of student 
success. In Artifact M, "Questions for Parents," researchers ask par- 
ents what sucess for their child would mean for them. Faculty also 
have developed a matrix that charts the desired outcomes for the 
PDS against possible data sources. Artifact N lists the six research 
questions across the top of the chart: measuring students' reading 
and bilingual progress and the school's image, as well as assessing 
attempts to broaden leadership, improve the student teaching pro- 
gram, and use data to improve writing instruction. On the vertical 
axis, they cite about a dozen data sources for each question. And while 
the acronyms and shorthand abbreviations will not provide specific 
detail to someone not familiar with the collaborative, the chart repre- 
sents a powerful example of creative use of multiple data sources to 
inform issues achievement and progress. 

ARTIFACT M 

Washington PDS/SJSU Questions for Parents 

1. Tell us about your family (and your community). How important is the Wash- 
ington School to your family? 

2. What is your son/daughter really good at doing? 

3. What does he or she need help with? 

4. Did you know there is a partnership between San Jose State University and 
Washington Elementary School called a Professional Development School? 

5. We are trying to help teachers do a better job of teaching students to read and 
write. Do you think that is important? Explain. 

6. Do you want your child to learn to read and write in Spanish or English or 
both? 

7. How well do you think you child can read/write now? 

8. Do you (or other relatives) help him/her learn to read at home? How? 

9. How can the school better assist your child? 

10. Do you have questions for us? 

Could I take some pictures of your home and family? 

Social Security # Date 

Assessing the Impacts of Professional Development Schools 

62 




ARTIFACT N 

Washington PDS/San Jose State University 

Data Sources for Research Questions 



BASRC 



NCATE 



1. Students’ 2. Students’ 3. School 

Reading Bilingual Image 

Progress Progress 



4. Broadening 5. Student 

Leadership Teaching 

Program 



6. Using Data 
to Improve 
Writing 
Instruction 



Sat 9 


ALA Program 


Media 

Examples 


MA Program 
Impact 


ST Focus 
Group 


Literacy Staff 
Development 


SABE 


End of Unit 
Assessments 


Physical- 
Dedication, 
Youth Center 


Action 

Research 

Examples 


MT Focus 
Group 


Al’s Interviews 


HB Tests 


SOLOM 


Parents Focus 
Group 


Marketplace 

Demos 


Classroom 

Observations 


Release Time 


Letter Names; 
Readiness 


QIA (for LAS 
oral) 


Governor’s 

Visit 


PDS Survey 


Lesson Plans 


Survey for 
Faculty 


Sight Words 


LAS O/R/W 


Academic 

Progress 


What can 1 
do? Dis. notes 


Pre-post 

Conferences 


Peer Visits 


Rhyme Lists; 

Phonics 

Survey 


Transition 

Moves 


District Notes 
of 

Recognition 


Level of 
Participation 


Graduate 

Survey 


Janis' MA 


ELIC/LL1FE 
Staff Dev. 
Records 


Regular 

Reading 

Progress 


Critical 
Friends Visit 


Design Team 
Meeting Tapes 


Orientation for 
Mts & Sts 


District Writing 
Assessment 


Technology: 
Acc. Reader/ 
Star 


Redesignation 
District’s R- 
30 Report 


SSC/SAC/ 

BAC Minutes/ 
Agenda 


Research 
Buddies 
Focus Group 




Janis’ Demos 


Student 

/Teacher 

Interviews 


Edythe’s MA 
ELD as 
regular prog. 


Family 

Literacy Days 


Grade Level 
Meetings ou 
Literacy 




MA Action Res. 
Presentations 


Perceptions & 
Realities 


Scholastic 
Exit Tests 


Res. Buddies 
Home Visits 


Principal 
Training on 
Assessment 




Beginning 
Teacher Meeting 
Notes 


Instructional 
Reading Level 
Changes 


David’s 
Interviews on 
Biting. Prog. 


Criteria for 
Not Choosing 
SFA 


Math Teacher 
Leadership 
(Paulette & 
Melitta) 




NCATE Self 
Study 


Janis’ Reading 
Level 

Assessment 




NCATE Visit 


PE Teacher 
Leadership 
(PQR Comp.) 




Literacy 

Learning 

Logs 


Reading 

ObsiRunn. 

Rees. 




Community 

Partnership 

Records 
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1. Students’ 

Reading 

Progress 



2. Students’ 

Bilingual 

Progress 



3. School 
Image 



4. Broadening 
Leadership 



5. Student 
Teaching 
Program 



6. Using Data 
to Improve 
Writing 
Instruction 



Reading 


Requests to 


Buddy 


Showcase 


Portfolios 


PDS 


District AccL 


Playground 


Matrix 


Redesign 


Instructional 


Community 


Observe rations 


Coalition 


DisL LA. 




Standards 




San Diego 


PE/Health 
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West Alabama Learning Coalition 

Like several other initiatives discussed in this chapter, the West 
Alabama Learning Coalition has designed a research approach that 
works across the collective but also works locally, for each partner- 
ship. Artifact O, " Assumptions Underlying the Evaluation Design" 
outlines the working assumptions guiding the Coalition's research 
efforts. The Coalition has a three-phase evaluation system, which 
maps onto the three stages of PDS development articulated by the 
PDS Standards Project. Artifact P outlines steps in the third phase of 
evaluation. 



ARTIFACT O 

Assumptions Underlying the Evaluation Design 

The underlying assumptions that have guided the design of our 
system, and which are potentially of value to others as they design 
systems of their own, are: 

1. While there must be a common set of indicators within an evalua- 
tion system, context must be a consideration in determining what 
should be assessed and how. Latitude must be given to enable the 
participants to focus on individual partnership goals and objectives, 
while at the same time recognizing the need for common standards 
and for equivalent data to be collected from each partnership within 
the Coalition. There must be a blending of flexibility and rigor in 
order to produce credible data on professional development schools. 

2. When creating systems of evaluation, one must remember that 
different audiences have different needs. Consequently, there should 
be a combination of both qualitative and quantitative measures within 
the evaluation design. 

3. The evaluation process should be ongoing and include efforts to 
document process initiatives; examine organizational, structural, cul- 
tural, and instructional changes; identify and measure intended out- 
comes; and consider impacts to date. 

4. At least one person should be designated as responsible for coor- 
dinating data collection and organization for each PDS site, and guide- 
lines should be developed to guide the process. This helps to clarify 
responsibility, offers consistency, and improves communication. 
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5. Evaluation systems should include all PDS participants in deter- 
mining goals, identifying data collection processes and materials, 
analyzing data, and using data to make decisions. Such involvement 
helps to ensure commitment to the effort and develops an under- 
standing and appreciation for the evaluative process. 

6. The evaluation process should encourage partnerships to acquaint 
themselves with and examine overall PDS purposes and general qual- 
ity standards. A goal of PDS evaluation should be to facilitate reflec- 
tion and dialogue about how well the PDS is accomplishing its pur- 
poses and measuring up against the set of established indicators. 

7. The evolving nature of the PDS must be considered within the 
design. Building in mechanisms for feedback loops is an important 
way to help each partnership develop and maintain their capacity 
for change. 

8. Involvement in the PDS evaluation process should focus upon 
building the individual and organizational capacity needed to trans- 
form systems, programs, and relationships. 
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ARTIFACT P 

West Alabama Learning Coalition 



Phase Three: Identification and Description of Outcomes 



Objectives 


To identify and describe the outcomes, both 
intended and unintended, of the PDS project 


Questions 


1. What evidence is there that the quality of 
preservice teacher education has been 
improved? 

2. What evidence is there that the PDS has 
helped to provide opportunities for continu- 
ing education for all? 

3. What evidence is there that collaborative 
inquiry was conducted? 

4. What evidence is there that the PDS 
provided an exemplary education for all 
students? 

5. What evidence is there that the PDS 
provided avenues for the development of 
collaborative learning communities? 

6. What other outcomes or unintended 
outcomes resulted from the PDS experience? 


Method / Strategies 


Surveys 
Focus Groups 
Interviews 


Data Sources 


All PDS participants 

Course assessments on preservice teachers 

Teaching portfolios 

Student grades and/or portfolios 

Artifacts such as letters of commendation or 

thanks 

Any other data brought forth or created 
through the research 


Who is Responsible? 


All PDS participants 
Evaluator responsible for encouraging 
research and conducting /analyzing focus 
groups 



Source: Reed et al. (1999) 
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A final artifact (Artifact Q) illustrates the ways some research 
initiatives have chosen to look at intermediate measures, organiza- 
tional changes and signs of institutionalization, as part of the process 
of assessing progress toward anticipanted outcomes for students, 
preservice educators, and others. Readers interested in this, drawn 
from Teitel, Reed and O'Connor (1998), may also wish to look at the 
Threshold Conditions of the PDS Standard Project of the National 
Council for the Accreditation of Teacher Education (Levine, 1998). 



1. Background information 

• size of teacher preparation program 

• relative proportion of students and faculty involved in PDS 

• roles of school and university-based faculty in PDS 

• changes in numbers or roles over the last few years 

• involvement of faculty from other university units 

2. What is happening? What are signs of institutional change at your site? 

A. At the college or university (partial list) 

• modification of courses or programs to better draw on practice 

• involvement of school-based faculty in planning and teaching courses 

• shifts in accountability and responsibility for student teacher evaluation 

• change in reward structures to support and encourage involvement with PDS 

• changes in attitudes at college about schools and the role of practice 

• different approaches to governance that draw in school-based personnel 

B. At the school (partial list) 

• involvement of college-based faculty in planning and teaching courses 

• increased sense of ownership of K-12 student learning by college faculty 

• greater school-based faculty responsibility for student teachers 

• changes in numbers of teachers and students involved in PDS 

• different approaches to governance that draw in college personnel 

• changes in reward structures to support and encourage involvement with PDS 

• increases in inquiry /reflective practice 

C. At the collaborative level (partial list) 

• changes in governance funding patterns 

• types of decisions being made by the Collaborative 

• number or types of people involved (from each institution) 

• new roles that span institutional boundaries 

• contracts, letters of agreement, mission statements (as evidence) 

• increased joint activities (research, study groups, etc.) 

3. What do you hope will happen? The checklist above may or may not 
address the institutional changes you hope to see. What specifically do you 
hope for in the context of your situation? 

4. What will it take to bring about the kind of changes you hope for? 

A. What do you see as the roadblocks? 

B. What is helping, or could help overcome those roadblocks? 

C. Who are the necessary players? 



ARTIFACT Q 

Institutionalization Focus Questions 



Source: Teitel, Reed, & O'Connor, 1998, pp. 55 
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